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DOCUMENT- IDENTIFIER : US 6766038 Bl 

TITLE: Apparatus and method for image processing 


Abstract Text (1) : 

The method for measuring speed is performed by dividing the time sequence image 
data A into long and narrow frames (step 401) . Next, the framed images and the time 
sequence image data B produced by the line scan camera B are correlated to obtain a 
moving time (step 402) . Next, a moving time of each framed image and a 
corresponding moving distance are obtained to compute a moving speed (step 403) . 
Next, scale of a specific shape pattern and the like is corrected using the moving 
speed obtained to match the scales in the two image sequences A and B to obtain a 
similarity degree (step 404) . Next, a threshold value is applied to the similarity 
degree to recognize the specific shape pattern, thereby recognizing that the images 
are produced by a moving vehicle. 

Brief Summary Text (3) : 

The present invention relates a method for measuring the length and a relative 
moving speed of an object moving with respect to a point of observation from time 
sequence image data produced by using line scan cameras and the like, and a pattern 
recognition method and apparatus based on the resulting image data. 

Brief Summary Text (6) : 

On the other hand, the following references are examples of the method for 
determining the speed and length of a moving object from images. "Traffic flow 
measurements using the double slit method" (reference 1, Road Traffic and 
Automobiles, The Institute of Electrical Engineers of Japan, Vol. RTA94-5, 1994) 
and "Traffic flow measurements using the double slit camera " (reference 2, The 
Institute of Electronics Engineers of Japan, Vol.26, No. 3, 1997). These methods are 
based on providing imaginary slits within the images recorded using a general 
purpose video -camera and obtaining spatial time images by linking the images 
between the slits. Two slits are placed in the images, so that the moving speed and 
the length of the object are measured by obtaining the time interval for the object 
to pass through the two slits. 

Brief Summary Text (9) : 

In the meantime, according to conventional video images, because the two slit 
planes are not parallel, the depth of field must be fixed first, so that the 
distance of movement between the actual slits can be determined separately. .Also, 
the general purpose video camera can only record at 30 frames/second, so that if 
the object is moving at high speeds, it is not possible to produce precision 
measurement of speed. Also, there is no known method of measuring the relative 
speed of an object using the images taken by placing a camera on another moving 
object. 

Brief Summary Text (19) : 

Also, in the present invention, there is no need for placing line image recording 
apparatus (line scanning camera and the like) to surround the object, and 
therefore, the arrangement of the apparatuses is facilitated compared with 
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using two units of line scan cameras . 
Detailed Description Text (26) : 

so that the time scales can be matched in the two image data. In the meantime, the 
scale in the line axis direction can vary depending on how far object is from the 
line scan cameras A recording the object. Therefore, the line scale of the template 
of the specific shape pattern in the line axis direction is increased or decreased 
to match the scales in the two images. After which, the degree of similarity is 
obtained by computing the difference value or correlation coefficient (refer to 
reference 3), and when a higher degree of similarity than a certain value of 
similarity is obtained, it is considered that a match has been obtained. In this 
embodiment, the template for the specific shape pattern was corrected to match the 
time scales, it is permissible to correct the time sequence image data, or to 
correct both the template and the time sequence image data. 

Detailed Description Text (27) : 

Next, the pattern is recognized using the degree of similarity, and the stationary 
vehicle is recognized (step 406) . By applying the threshold processing technique to 
the degree of similarity obtained in step 405, the wheel represented by the 
specific shape pattern is recognized. Therefore, if a wheel is detected within the 
time sequence image data, it means that the stationary object is a stationary 
vehicle. Accordingly, a stationary object can be recognized using the time sequence 
image data produced from the two line scan cameras . 

Detailed Description Text (30) : 

Measurements of speed and length, a method for recognizing a pattern and embodied 
examples of the apparatus in Embodiment 2 will be explained in the following. In 
this embodiment, a measuring point having two units of line scan cameras, as an 
example of a plurality of line scan cameras, are disposed in a fixed point of 
observation to perform speed and length measurements and pattern recognition of 
vehicles moving along a given track. The measuring apparatus used in this 
embodiment is the same as the one used in Embodiment 1 (refer to FIG. 1) . 

Detailed Description Text (31) : 

FIG. 7 shows an example of the apparatus for performing speed and length 
measurements and pattern recognition in Embodiment 2, and 701 refers to a moving 
vehicle; 702 to a line scan camera A; 703 to a line scan camera B, 704 to a 
measuring apparatus. 

Detailed Description Text (32) : 

FIG. 8 shows a top view of the measuring system shown in FIG. 7, and 801 refers to 
a moving vehicle; 802 to the moving direction; 803 to a line scan camera A; 804 to 
a line scan camera B; 805 to an angle .theta.; 806 to a line axes separation 
distance L; and 807 to a moving distance Lr. The moving direction of the moving 
vehicle 801 is defined as the moving direction 802, and the line scan camera A803 
and the line scan camera B804 are disposed at an angle .theta. with respect to the 
moving direction 802. In this arrangement, the true distance Lr807 of the vehicle 
moving along the given track between the line axes is given the expression below. 

Detailed Description Text (35) : 

First, an image pattern that includes the moving vehicle 801 is extracted from time 
sequence images 911, 913 recorded at a given scanning timing using the line scan 
camera A803 synchronized to the line scan camera B804 (step 901) . Images without 
the moving vehicles are recorded beforehand using the line scan camera A803 and the 
line scan camera B804, and are accumulated in the image accumulation section 107 as 
background image A912 and B914. 

Detailed Description Text (36) : 

FIG. 10 shows examples of the time sequence image data recorded by the line scan 
cameras, and 1001 refers to time sequence image data A recorded by the line scan 
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by synchronizing the two cameras at a constant timing determined by a line scanning 
cycle. 

Detailed Description Text (7) : 

The line scan cameras used in this embodiment are a monochromatic cameras having 8- 
bit pixels of 256 gradations each, and the number of pixels comprising each line is 
2024 pixels and each line is scanned at a cycling time of 100 .mu.s. 

Detailed Description Text (8) : 

FIG. 2 shows a measurement method, a pattern recognition method and an example of a 
measurement system using the measuring apparatus, and 201 represents a stationary 
vehicle 1; 202a stationary vehicle 2; 203a moving vehicle; 204 the moving 
direction; 205 a line scan camera A installed on the moving vehicle 203; and 206 a 
line scan camera B installed on the moving vehicle 203. 

Detailed Description Text (9) : 

FIG. 3 is a top view of the measuring system shown in FIG. 2, and 301 represents 
the moving vehicle; 302 the moving direction; 303a line scan camera A; 304a line 
scan camera B; 305 an angle .theta.; 306 a distance L between the line axes (line 
axes spacing distance L) ; 307a moving distance Lr of the moving vehicle; 308 a 
stationary vehicle 1; and 309 a stationary vehicle 2. The direction of the moving 
vehicle 301 is designated by 302, and it is assumed that the line scan camera A303 
and the line scan camera B304 are inclined at an angle .theta. 305 with respect to 
the moving direction 302. Then, the actual moving distance Lr 307 of the stationary 
vehicle between the line axes is obtained by an expression: 

Detailed Description Text (12) : 

First, time sequence image data A411 recorded by the line scan camera A303 are 
divided into long frames having a narrow width (step 401) . FIG. 5 shows an example 
of the time sequence image data and the framed images, and 501 refers to the time 
sequence image data A recorded by the line scan camera A, 502 refers to time 
sequence image data B recorded by the line scan camera B, 503 to a narrow 
rectangular framed image in the data A501, 504 to a matched region found in the 
data B302, 505 to a frame width W and 506 to a moving time interval Tv. Here, it is 
preferable that the number of lines along the time axis of a framed image be about 
10. about. 20 lines, and since the number of pixels per line is 2,048, the number of 
pixels in one framed image is about 20, 480 . about . 40, 960 . 

Detailed Description Text (13) : 

Next, the framed image 503 is correlated with the time sequence image data B412 
recorded by the line scan camera B304, and a size of a respective time displacement 
506 of the object is computed (step 402) . It can be seen in FIG. 5 that the 
stationary vehicle crosses the viewing front of the line scan camera B304 before 
that of the line scan camera A303, so that the image region 504 in the time 
sequence image data B that matched the framed image 503 is found near the beginning 
portion of the time axis 507. Therefore, the degree of similarity is obtained by 
restricting the search to only the negative direction on the time axis 507 to 
compute the difference value and the correlation coefficient (refer to reference 3, 
"Image Processing Handbook", Shokodo, 1987) to find the matched region 504 to 
compute a size of the time displacement 506 of the object. 

Detailed Description Text (19) : 

Next, using the time interval of the framed images and the corresponding moving 
distance, a moving speed is obtained (step 403) . Specifically, the time 
displacement, in other words, the moving time interval Tv is obtained from the 
following expression using the number of moving pixels P and the recording time 
interval Tc of the line scan camera . 

Detailed Description Text (22) : 

As described above, it is possible to determine a moving speed of an object by 
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obtain chronological change 1602 in the amount of image change. Designating the 
number of pixels in one line by m, the brightness of each point in the line at time 
t by I(t, i), and the line scanning interval by t.sub.O, the above-mentioned 
chronological change e(t) is given by the following expression. ##EQU3## 

Detailed Description Text (52): 

Threshold processing of the amount of change using the threshold value 1604 enables 
to determine an image change point 1603. Similar processing is performed on the 
time sequence image data B1512 recorded by the line scan camera B to obtain an 
image change point. 

Detailed Description Text (53) : 

Next, the image change points are correlated and the time displacement values are 
computed. As in this embodiment, when the moving direction of the moving vehicle is 
given, the object crosses the line scan camera A first and then the line scan 
camera B next. Therefore, an image change point is detected in the time sequence 
image data A, and an image change point obtained immediately afterward from the 
time sequence image data B can be designated as the starting point of the image 
pattern of the moving vehicle. Even when the moving direction of the vehicle is 
unknown, an image change point nearby can be designated as the starting point or an 
ending point of the image pattern of the moving vehicle. When the correlation is 
completed, moving velocity of the moving object can be computed from the moving 
time interval and the actual distance of move as in Embodiment 2. 

Detailed Description Text (56): 

Measurements of speed, length and a method recognizing a pattern and embodied 
examples of the apparatus in Embodiment 3 will be explained in the following. In 
this embodiment, a measuring point having two units of line scan cameras, as an 
example of a plurality of line scan cameras, which are disposed in a fixed point of 
observation to perform speed and pattern recognition of moving vehicles. The 
measuring apparatus used in this embodiment is the same as the one used in 
Embodiments 2, 3 (refer to FIGS. 7, 8). 

Detailed Description Text (59) : 

FIG. 18 shows an example of the chronological change in the amount of image change 
between the time sequence image data and the background image, and 1801 refers to 
the time sequence image data A recorded by the line scan camera A; 1802 to a 
background image A; 1803 to a chronological change in the amount of image change 
from the background image; 1804 to an image change point (starting point); 1805 to 
an image change point (ending point); 1806 to a threshold value for the amount of 
change; and 1807 to the time axis. Because the observation point is fixed in this 
embodiment, when there is no moving object, there is hardly any chronological 
change in the image. On the other hand, a large change takes place in the image 
when the moving vehicle begins to pass. Therefore, by performing difference value 
computation or correlation coefficient computation (refer to reference 3), it is 
possible to obtain chronological change 1803 in the amount of image change. 
Threshold processing of the amount of change using the threshold value 1806 enables 
to determine an image change point (starting point) 1804 and an image change point 
(ending point) 1805. Similar processing is performed on the time sequence image 
data B1713 recorded by the line scan camera B to obtain an image change point. 

Detailed Description Text (60) : 

Next, the image change points are correlated and the time displacement values are 
computed. In this embodiment, the starting point and the ending point of image 
change are considered as a combined pair of image change points. As in this 
embodiment, when the moving direction of the moving vehicle is given, the object 
crosses the line scan camera A first and then the line scan camera B next. 
Therefore, an image change point is detected in the time sequence image data A, and 
an image change point obtained immediately afterward from the time sequence image 
data B can be designated as the combined pair of starting point and the ending 
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camera A; 1002 to time sequence image data B recorded by the line scan camera B; 
1003 to a background image A of the line scan camera A; 1004 to a background image 
B of the line scan camera B; and 1005 to the time axis. Background images A1003, 
B1004 are images having a width comprised by one line or several lines. Because the 
observation point is stationary in this embodiment, images such as those shown by 
time sequence image data A1001 and B1002 formed by the moving vehicle 801 
superimposed on the contiguous background images are obtained. Therefore, by 
performing difference value computation involving the background image or 
correlation coefficient computation (refer to reference 3) for every one line of 
several lines, it is possible to perform threshold computation and labeling 
processing (refer to reference 3) using the values thus obtained, thereby enabling 
to extract regions different from the background, that is, an image pattern 
containing the moving object. However, the background images can change depending 
on the illumination conditions, and therefore, background images may in some cases 
be renewed as necessary. The foregoing expression (1) or (2) can be used to compute 
the above-mentioned difference value or correlation coefficient. 

Detailed Description Text (39) : 

Next, the moving speed is obtained from the value of the time displacement of the 
image patterns and the corresponding moving distance (step 903) . Specifically, the 
value of the time displacement, that is, the moving time interval Tv is obtained 
according to the number of moving pixels P of the image pattern in the time 
sequence image data and the recording time interval Tc of the line scan camera 
using the following expression. 

Detailed Description Text (41) : 

As described above, it is possible to determine a moving speed of an object by 
using two units of line scan cameras . 

Detailed Description Text (46) : 

Next, using the degree of similarity, the pattern is recognized and the moving 
vehicle is identified (step 906). Therefore, by applying the threshold processing 
technique to the degree of similarity obtained in step 905, the wheel represented 
by the specific shape pattern is recognized. Therefore, if a wheel is detected 
within time sequence image data, it means that the moving object is a moving 
vehicle. Accordingly, a moving object can be recognized using the time sequence 
image data produced from two line scan cameras . 

Detailed Description Text (48) : 

Measurements of speed and a method for recognizing a pattern and embodied examples 
of the apparatus in Embodiment 3 will be explained in the following. In this 
embodiment, a measuring point having two units of line scan cameras, as an example 
of a plurality of line scan cameras, are disposed in a fixed point of observation 
to perform speed and pattern recognition of moving vehicles. The measuring 
apparatus used in this embodiment is the same as the one used in Embodiment 2 
(refer to FIGS. 7, 8) . 

Detailed Description Text (51) : 

First, an image change point on the time axis is detected from the time sequence 
image data 1511, 1512 (step 1501) . Here, the time sequence image data are obtained 
as in Embodiment 2. FIG. 16 shows a graph of an example of the chronological change 
in the time sequence image data, and 1601 refers to the time sequence image data A 
recorded by the line scan camera A; 1602 is graph to show the chronological change 
in the amount of change between two consecutive 1-line images; 1603 to an image 
change point; 1604 to a threshold value of the change; and 1605 to the time axis. 
Because the observation point is fixed in this embodiment, when there is no moving 
object, there is hardly any chronological change in the image. On the other hand, a 
large change takes place in the image when the moving vehicle begins to pass a 
detection point. Therefore, by performing difference value computation or 
correlation coefficient computation (refer to reference 3), it is possible to 
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conventional phototubes and the like. In contract to using the phototubes, image 
data of the object can be accumulated so that the object can be identified after 
the measurements. 

Brief Summary Text (20) : 

Also, by using line image recording apparatus, it is possible to provide highspeed 
recording of a moving object at about 1000 times the speed of conventional video 
cameras, so that moving speed of a fast moving object can be determined. Unlike the 
case of using video cameras, many line image recording apparatuses can be 
positioned in parallel to enable speed measurements without being affected by 
problems related to the distance from the cameras . 

Drawing Description Text (6) : 

FIG. 5 is an illustration of an example of the output image from a line scan camera 
in Embodiment 1 . 

Drawing Description Text (11) : 

FIG. 10 shows examples of the output images from line scan cameras in Embodiment 2. 
Drawing Description Text (15) : 

FIG. 14 show side views of examples of the output images from a line scan camera in 
Embodiment 2 . 

Drawing Description Text (17) : 

FIG. 16 is an example of the output image from the line scan camera in Embodiment 3 
and a graph showing the amount of change in the image. 

Drawing Description Text (19) : 

FIG. 18 is an example of the output image from the line scan camera in Embodiment 4 
and a graph showing the amount of change in the image. 

Drawing Description Text (26) : 

FIG. 25 is an illustrated view of an example of the output from the line scan 
camera in Embodiment 5 . 

Drawing Description Text (28): 

FIG. 27 is an example of the output image from the line scan camera in Embodiment 5 
and a graph showing the amount of change in the image. 

Detailed Description Text (5) : 

Speed measurements and pattern recognition method and embodied examples of the 
apparatus in Embodiment 1 will be explained. In this embodiment, two units of line 
scan cameras, as an example of a plurality of line scan cameras, are used in the 
system and are mounted on a moving vehicle to perform image processing so that the 
speed of the moving vehicle can be measured and the image of a stationary object 
represented by a stationary vehicle can be recognized. In other words, the relative 
speed of the object moving relatively to the system along a given track can be 
measured. 

Detailed Description Text (6): 

FIG. 1 shows an apparatus for performing the speed measurement and pattern 
recognition method in Embodiment 1. In to the information processing apparatus 101 
shown in FIG. 1, a memory device 103, a display device 104 such as image monitor 
and a measuring device 105 are connected thereto through a bus line 102. The memory 
device 103 is provided, in its interior, with a measurement results accumulation 
section 106 and an image accumulation section 107. Also, inside the measuring 
device 105 are provided a line scan camera A108, a line scan camera B109 and a 
synchronizing device 110. In this case, the synchronizing device 110 is represented 
by a pulse generator. The time image data from the line scan cameras are recorded 
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FOREIGN PATENT DOCUMENTS 


FOREIGN- PAT-NO PUBN-DATE COUNTRY CLASS 

2-127877 May 1990 JP 

WO 82/03712 October 1982 WO 

OTHER PUBLICATIONS 

English Abstract 2-127877 . (A) , May 16, 1990, for "Electronic Still Camera Provided 
With Fisheve Lens " for Japanese Application No. 63-281550. 
"Fooling the Eye", Suzanne Oliver, Forbes, Jan. 16, 1995 (page 94) . 
Exhibit 4 -"The Omnigraph, Omnidirectional Spherical Photography" and 
"Omnigraphics, second report," Richard J. Felix, 1979 f s. 

Exhibits 6, 26, 29, 30, 32-35 — Omnigraphics course materials, California State 
University —Richard J. Felix —1974 to 1994. 

Exhibit 17 — "Multiplex Video Display", Richard J. Felix, Feb. 7, 1994. 
ART-UNIT: 264 

PRIMARY-EXAMINER: Lee; Michael H. 
ATTY-AGENT-FIRM: Banner & Witcoff, Ltd. 


ABSTRACT : 

A method for capturing and directly scanning a rectilinear imaging element using a 
non-linear scan is incorporated into a single chip comprising at least a sensor 
array and an MSD. The method directly addresses each picture element of an analog 
image captured with an imaging device having either a partial spherical field of 
view or a conventional two-dimensional field of view. An image transform processor 
is used to process the captured image depending upon the particular portion of 
interest of the image. In the case of a non -linear scan, the image transform 
processor is provided with the capability of geometrically filtering the portion of 
interest of the captured image such that a two-dimensional, undistorted image is 
displayed at the monitor. A CMOS active pixel image sensor (APS) or Charge 
Injection Diode (CID) camera array are used to capture the image to be scanned. The 
image transform processor of the present invention is a Mixed-signal Semiconductor 
Device (MSD) . The image transform processor corrects any predetermined distortion 
introduced by the image sensor array. 

17 Claims, 9 Drawing figures 
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point of the image pattern of the moving vehicle. Even when the moving direction of 
the vehicle is unknown, an image change point pair nearby can be designated as the 
starting point and an ending point of the image pattern of the moving vehicle. When 
the correlation is completed, moving velocity of the moving object can be computed 
from the moving time interval and the actual distance of move as in Embodiment 2. 

Detailed Description Text (62) : 

In the above Embodiments 2, 3 and 4, the side view of the object was used, but as 
shown in FIGS. 19, 20, the method is applicable to a case of images recorded by the 
line scan cameras A1902, 2003 and the line scan cameras B1903, 2004 which are 
disposed on the upper section of the moving vehicles 1901, 2001. Also, as shown in 
FIGS. 21, 22, to accumulate the processing results, line scan cameras A2102, 2202 
and the line scan camera A303 line scan cameras B2103, 2203 are synchronized with 
the area scan cameras 2104, 2204, and the stationary image and video images 
recorded by the moving objects (moving vehicles 2101, 2201) may be displayed or 
recorded. Also, two units of line scan cameras were used, but more than 2 units of 
line scan cameras may be provided. 

Detailed Description Text (64) : 

Next, an application of the present method of speed measurement to a speeding 
vehicle monitoring system will be explained. In this embodiment, two units of line 
scanning cameras assigned to the respective observation points are fixed, and speed 
measurement of the moving vehicle and license plate recognition are performed. 

Detailed Description Text (65) : 

FIG. 23 shows an example of the speeding vehicle monitoring system in the present 
embodiment. The line scan camera A3102, a line scan camera B3103 and an infrared 
illumination 3104 are disposed above the gantry so as to enable a speeding car to 
be recorded from above. The line axes of the line scan camera A3102 and the line 
scan camera B3103 are disposed parallel to each other, and are synchronized so as 
to record at a constant line scanning cycle. Also, the line scan camera A3102 and 
the line scan camera B3103 are each provided with a polarizing filter for 
suppressing scattered reflection from the front glass and to enable to record an 
image of the driver, as well as a visible light blocking filter (infrared 
transmissive filter) to enable to record the images any time of the day or night. 

Detailed Description Text (66) : 

FIG. 24 shows a side view of the system shown in FIG. 23, and the line scan camera 
A3203 and the line scan camera B3204 and the infrared illumination 3208 are 
oriented downward at and angle .theta. with respect to the ground. In such a case, 
the moving distance Lr of the moving vehicle between the line scan camera A and the 
line scan camera B is given by the following relation. 

Detailed Description Text (67) : 

FIG. 25 shows an example of the time sequence images obtained by the line scan 
camera A and the line scan camera B. As shown in this diagram, the apparatus is 
capable of recording a driver and a license plate by placing the line scan cameras 
so as to view from the top to the bottom. 

Detailed Description Text (69): 

First, time sequence images A produced by the line scan camera A and time sequence 
images produced by line scan camera B are obtained, and detect an image change 
point on the time axis from each time sequence image data (step 3401) . 

Detailed Description Text (71) : 

Next, image change points in time sequence images A and B are correlated to each 
other and moving time intervals are computed (step 3402) . When the moving direction 
of the moving vehicle is already determined, the vehicle crosses the cameras in the 
order of line scan camera A and line scan camera B. Therefore, an image change 
point is detected from the time sequence image data A, and an image change point in 
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the time sequence image data B, that follows immediately the images in camera A, 
can be identified as the starting point. Therefore, the moving distance Lr between 
the two line scan cameras and the moving time Tv are as shown in FIG. 28. 

Detailed Description Text (74) : 

FIG. 29 shows examples of the time sequence image data obtained from a fast moving 
vehicle and a slow moving vehicle. As shown in FIG. 29, the characters on the 
license plate are extended when the speed is low and are compressed when the speed 
is fast. Therefore, by dividing the time axis in each time sequence image data by 
the moving speed, the height of the characters on the license plate can be 
normalized. If the characters can be normalized, the license plate of the moving 
vehicle can be identified without being affected by the moving speed, using the 
conventional license plate recognition technique (refer to reference 4, Character 
Recognition in Scene Images, Society of Manufacturing Engineers, 1989) based on the 
degree of similarity. In the present example, because there are two image data from 
line scan camera A and line scan camera B for identification purposes, license 
plate can be checked against each image to improve the accuracy of identification. 

Detailed Description Text (77) : 

It should be noted, in the explanations provided above, that reference numerals 
108, 205, 303, 702, 803, 1202, 1303, 1902, 2003, 2102, 2202, 3102 refer to the same 
time sequence image data A. Similarly, reference numerals 109, 206, 304, 703, 804, 
1203, 1304, 1903, 2004, 2103, 3303, 3103, 3204 refer to the same line scan camera 
B. Similarly, reference numerals 701, 801, 1201, 1301, 1901, 2001, 2101, 2201 refer 
to the object for measurement. 
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